[SPARK-59438][SQL][TESTS] Add planner-stage test coverage for ASOF JOIN sort-merge operator - #58742
Open
LukaZdravic wants to merge 6 commits into
Open
LukaZdravic wants to merge 6 commits into
LukaZdravic wants to merge 6 commits into
Conversation
…IN sort-merge operator Adds two planner-stage tests for the ASOF JOIN sort-merge path, both mirroring existing sibling-join test patterns: - `PlannerSuite`: a no-equi-key `AsOfJoin` requires a single partition on both sides (the `AllTuples` branch of `SortMergeAsOfJoinExec`'s `requiredChildDistribution`), verified by running `EnsureRequirements` and asserting a single-partition shuffle on each side. Follows the existing `SortMergeJoinExec` distribution tests (SPARK-24495 / SPARK-27485). - `SortMergeAsOfJoinSuite`: a null-safe (`<=>`) equi-key in `ON` is routed to the residual condition rather than treated as an equi-key, so null keys on both sides do match -- the counterpart to the existing `EqualTo` case where null keys never match. Mirrors `InnerJoinSuite`'s "inner join, null safe". Test-only change; no production code is modified.
There was a problem hiding this comment.
🟢 Approval recommended
Test-only changes with no unresolved issues.
Pull request overview
Adds test-only planner and execution coverage for ASOF JOIN behavior.
Changes:
- Tests single-partition requirements for no-equi-key joins.
- Tests null-safe conditions matching null keys.
File summaries
| File | Summary |
|---|---|
sql/core/src/test/scala/org/apache/spark/sql/SortMergeAsOfJoinSuite.scala |
Adds null-safe ASOF JOIN coverage. |
sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala |
Adds distribution requirement coverage. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…l path The null-safe (<=>) as-of test gave each left row one null candidate, so it could not tell "keep closest" from "keep first" on the residual path. Reuse the EqualTo sibling's data (only the ON predicate differs) and add a null-key left row at ts=10 with two null candidates (ts=3, ts=8). The scanner must keep the closest (ts=8), so a keep-first regression now fails.
vranes
reviewed
Sep 14, 2026
…qui-key The null-safe (<=>) as-of test checked only output rows, which pass whether <=> is routed to the residual condition or treated as a null-safe equi-key. Inspect the executed plan and assert the SortMergeAsOfJoinExec has empty equi-keys, so the test verifies the routing its comment describes.
…t comments Address review feedback: - Add the equi-key counterpart to the no-equi-key distribution test, asserting each side hash-partitions on the key rather than shuffling to one partition. - Assert requiredChildOrdering in both distribution tests (keys then sort exprs). - Shorten the <=> and distribution test comments to one line each.
…n both tests The distribution was checked only through EnsureRequirements, but the if (leftKeys.isEmpty) AllTuples else ClusteredDistribution branch is the behavior under test. Assert it directly: AllTuples with no equi-keys, and ClusteredDistribution on the key with equi-keys.
… just its type The equi-key test asserted the shuffle outputPartitioning isInstanceOf HashPartitioning, a type-only check. Assert its expressions equal Seq(exprC) so a hash on a different column fails the test.
LukaZdravic
marked this pull request as ready for review
September 14, 2026 13:43
uros-b
approved these changes
Sep 14, 2026
Member
|
Thank you @LukaZdravic and @vranes! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Test-only change. It adds planner-stage (physical-planning) coverage for the ASOF JOIN
sort-merge operator. No production code changes.
PlannerSuite: two tests build aSortMergeAsOfJoinExec, runEnsureRequirements, andcheck
requiredChildDistribution:AllTuplesarm).ClusteredDistributionarm).SortMergeAsOfJoinSuite: one test checks that a null-safe equi-key (<=>) inONbecomes a residual condition, not an equi-key. Rows whose key is NULL on both sides then match. This is the counterpart to the
EqualTocase (NULL keys never match) and mirrorsInnerJoinSuite's "inner join, null safe" test.No production code is changed.
Why are the changes needed?
The
AsOfJoinSelectionstrategy andSortMergeAsOfJoinExecoperator were tested end-to-end for result correctness, but two planner-stage behaviors had no direct test:requiredChildDistribution(single partition when there are noequi-keys, hash-partition on the key otherwise), and
EqualNullSafe(<=>) inONto the residual condition insteadof an equi-key.
Sibling join operators cover the same behaviors. This brings ASOF JOIN to the same bar.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
New unit tests. Ran
PlannerSuiteandSortMergeAsOfJoinSuitelocally (via Maven); all tests pass.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code